-
Notifications
You must be signed in to change notification settings - Fork 992
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[main] Update dependencies from dotnet/runtime #12631
[main] Update dependencies from dotnet/runtime #12631
Conversation
…1211.9 Microsoft.Internal.Runtime.WindowsDesktop.Transport , Microsoft.NET.Sdk.IL , Microsoft.NETCore.App.Ref , Microsoft.NETCore.App.Runtime.win-x64 , Microsoft.NETCore.ILAsm , Microsoft.NETCore.ILDAsm , Microsoft.NETCore.Platforms , Microsoft.Win32.Registry.AccessControl , Microsoft.Win32.SystemEvents , runtime.win-x64.Microsoft.NETCore.ILAsm , runtime.win-x86.Microsoft.NETCore.ILAsm , System.CodeDom , System.ComponentModel.Composition , System.ComponentModel.Composition.Registration , System.Configuration.ConfigurationManager , System.Data.Odbc , System.Data.OleDb , System.Diagnostics.EventLog , System.Diagnostics.PerformanceCounter , System.DirectoryServices , System.DirectoryServices.AccountManagement , System.DirectoryServices.Protocols , System.Formats.Nrbf , System.IO.Hashing , System.IO.Packaging , System.IO.Ports , System.Management , System.Reflection.Context , System.Reflection.MetadataLoadContext , System.Resources.Extensions , System.Runtime.Caching , System.Runtime.Serialization.Formatters , System.Security.Cryptography.Pkcs , System.Security.Cryptography.ProtectedData , System.Security.Cryptography.Xml , System.Security.Permissions , System.ServiceModel.Syndication , System.ServiceProcess.ServiceController , System.Speech , System.Text.Encoding.CodePages , System.Text.Encodings.Web , System.Text.Json , System.Threading.AccessControl , System.Windows.Extensions , VS.Redist.Common.NetCore.SharedFramework.x64.10.0 From Version 10.0.0-alpha.1.24611.1 -> To Version 10.0.0-alpha.1.24611.9
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Go, you big red fire engine!
…1213.1 Microsoft.Internal.Runtime.WindowsDesktop.Transport , Microsoft.NET.Sdk.IL , Microsoft.NETCore.App.Ref , Microsoft.NETCore.App.Runtime.win-x64 , Microsoft.NETCore.ILAsm , Microsoft.NETCore.ILDAsm , Microsoft.NETCore.Platforms , Microsoft.Win32.Registry.AccessControl , Microsoft.Win32.SystemEvents , runtime.win-x64.Microsoft.NETCore.ILAsm , runtime.win-x86.Microsoft.NETCore.ILAsm , System.CodeDom , System.ComponentModel.Composition , System.ComponentModel.Composition.Registration , System.Configuration.ConfigurationManager , System.Data.Odbc , System.Data.OleDb , System.Diagnostics.EventLog , System.Diagnostics.PerformanceCounter , System.DirectoryServices , System.DirectoryServices.AccountManagement , System.DirectoryServices.Protocols , System.Formats.Nrbf , System.IO.Hashing , System.IO.Packaging , System.IO.Ports , System.Management , System.Reflection.Context , System.Reflection.MetadataLoadContext , System.Resources.Extensions , System.Runtime.Caching , System.Runtime.Serialization.Formatters , System.Security.Cryptography.Pkcs , System.Security.Cryptography.ProtectedData , System.Security.Cryptography.Xml , System.Security.Permissions , System.ServiceModel.Syndication , System.ServiceProcess.ServiceController , System.Speech , System.Text.Encoding.CodePages , System.Text.Encodings.Web , System.Text.Json , System.Threading.AccessControl , System.Windows.Extensions , VS.Redist.Common.NetCore.SharedFramework.x64.10.0 From Version 10.0.0-alpha.1.24611.1 -> To Version 10.0.0-alpha.1.24613.1
ed226b4
…1213.10 Microsoft.Internal.Runtime.WindowsDesktop.Transport , Microsoft.NET.Sdk.IL , Microsoft.NETCore.App.Ref , Microsoft.NETCore.App.Runtime.win-x64 , Microsoft.NETCore.ILAsm , Microsoft.NETCore.ILDAsm , Microsoft.NETCore.Platforms , Microsoft.Win32.Registry.AccessControl , Microsoft.Win32.SystemEvents , runtime.win-x64.Microsoft.NETCore.ILAsm , runtime.win-x86.Microsoft.NETCore.ILAsm , System.CodeDom , System.ComponentModel.Composition , System.ComponentModel.Composition.Registration , System.Configuration.ConfigurationManager , System.Data.Odbc , System.Data.OleDb , System.Diagnostics.EventLog , System.Diagnostics.PerformanceCounter , System.DirectoryServices , System.DirectoryServices.AccountManagement , System.DirectoryServices.Protocols , System.Formats.Nrbf , System.IO.Hashing , System.IO.Packaging , System.IO.Ports , System.Management , System.Reflection.Context , System.Reflection.MetadataLoadContext , System.Resources.Extensions , System.Runtime.Caching , System.Runtime.Serialization.Formatters , System.Security.Cryptography.Pkcs , System.Security.Cryptography.ProtectedData , System.Security.Cryptography.Xml , System.Security.Permissions , System.ServiceModel.Syndication , System.ServiceProcess.ServiceController , System.Speech , System.Text.Encoding.CodePages , System.Text.Encodings.Web , System.Text.Json , System.Threading.AccessControl , System.Windows.Extensions , VS.Redist.Common.NetCore.SharedFramework.x64.10.0 From Version 10.0.0-alpha.1.24611.1 -> To Version 10.0.0-alpha.1.24613.10
…1216.1 Microsoft.Internal.Runtime.WindowsDesktop.Transport , Microsoft.NET.Sdk.IL , Microsoft.NETCore.App.Ref , Microsoft.NETCore.App.Runtime.win-x64 , Microsoft.NETCore.ILAsm , Microsoft.NETCore.ILDAsm , Microsoft.NETCore.Platforms , Microsoft.Win32.Registry.AccessControl , Microsoft.Win32.SystemEvents , runtime.win-x64.Microsoft.NETCore.ILAsm , runtime.win-x86.Microsoft.NETCore.ILAsm , System.CodeDom , System.ComponentModel.Composition , System.ComponentModel.Composition.Registration , System.Configuration.ConfigurationManager , System.Data.Odbc , System.Data.OleDb , System.Diagnostics.EventLog , System.Diagnostics.PerformanceCounter , System.DirectoryServices , System.DirectoryServices.AccountManagement , System.DirectoryServices.Protocols , System.Formats.Nrbf , System.IO.Hashing , System.IO.Packaging , System.IO.Ports , System.Management , System.Reflection.Context , System.Reflection.MetadataLoadContext , System.Resources.Extensions , System.Runtime.Caching , System.Runtime.Serialization.Formatters , System.Security.Cryptography.Pkcs , System.Security.Cryptography.ProtectedData , System.Security.Cryptography.Xml , System.Security.Permissions , System.ServiceModel.Syndication , System.ServiceProcess.ServiceController , System.Speech , System.Text.Encoding.CodePages , System.Text.Encodings.Web , System.Text.Json , System.Threading.AccessControl , System.Windows.Extensions , VS.Redist.Common.NetCore.SharedFramework.x64.10.0 From Version 10.0.0-alpha.1.24611.1 -> To Version 10.0.0-alpha.1.24616.1
…1216.9 Microsoft.Internal.Runtime.WindowsDesktop.Transport , Microsoft.NET.Sdk.IL , Microsoft.NETCore.App.Ref , Microsoft.NETCore.App.Runtime.win-x64 , Microsoft.NETCore.ILAsm , Microsoft.NETCore.ILDAsm , Microsoft.NETCore.Platforms , Microsoft.Win32.Registry.AccessControl , Microsoft.Win32.SystemEvents , runtime.win-x64.Microsoft.NETCore.ILAsm , runtime.win-x86.Microsoft.NETCore.ILAsm , System.CodeDom , System.ComponentModel.Composition , System.ComponentModel.Composition.Registration , System.Configuration.ConfigurationManager , System.Data.Odbc , System.Data.OleDb , System.Diagnostics.EventLog , System.Diagnostics.PerformanceCounter , System.DirectoryServices , System.DirectoryServices.AccountManagement , System.DirectoryServices.Protocols , System.Formats.Nrbf , System.IO.Hashing , System.IO.Packaging , System.IO.Ports , System.Management , System.Reflection.Context , System.Reflection.MetadataLoadContext , System.Resources.Extensions , System.Runtime.Caching , System.Runtime.Serialization.Formatters , System.Security.Cryptography.Pkcs , System.Security.Cryptography.ProtectedData , System.Security.Cryptography.Xml , System.Security.Permissions , System.ServiceModel.Syndication , System.ServiceProcess.ServiceController , System.Speech , System.Text.Encoding.CodePages , System.Text.Encodings.Web , System.Text.Json , System.Threading.AccessControl , System.Windows.Extensions , VS.Redist.Common.NetCore.SharedFramework.x64.10.0 From Version 10.0.0-alpha.1.24611.1 -> To Version 10.0.0-alpha.1.24616.9
We are observing access violations / ExecutionEngineExceptions in our tests with this update. I tried to investigate further by debugging our
These are the commits included in the first dependency update dotnet/runtime@aa9cd3b...c646425 but it is not obvious to me what may have caused this behavior. @ericstj are you aware of any changes that could have caused this or who can help investigate? |
@lonitra any chance you have a dump? Of those commits identified, the one that seems the most likely is dotnet/runtime@69d8076 cc @VSadov @jkotas. |
I have confirmed that it is caused by this change. I am reverting it in dotnet/runtime#110801 The crash repros fairly reliably for me locally. It is not obvious what the problem is. |
My guess - either winforms scenarios use FLS in a conflicting manner, or something that we do on thread detach in CoreCLR is somehow dependent on holding the loader lock (that would be a bit inconvenient). |
…1217.3 Microsoft.Internal.Runtime.WindowsDesktop.Transport , Microsoft.NET.Sdk.IL , Microsoft.NETCore.App.Ref , Microsoft.NETCore.App.Runtime.win-x64 , Microsoft.NETCore.ILAsm , Microsoft.NETCore.ILDAsm , Microsoft.NETCore.Platforms , Microsoft.Win32.Registry.AccessControl , Microsoft.Win32.SystemEvents , runtime.win-x64.Microsoft.NETCore.ILAsm , runtime.win-x86.Microsoft.NETCore.ILAsm , System.CodeDom , System.ComponentModel.Composition , System.ComponentModel.Composition.Registration , System.Configuration.ConfigurationManager , System.Data.Odbc , System.Data.OleDb , System.Diagnostics.EventLog , System.Diagnostics.PerformanceCounter , System.DirectoryServices , System.DirectoryServices.AccountManagement , System.DirectoryServices.Protocols , System.Formats.Nrbf , System.IO.Hashing , System.IO.Packaging , System.IO.Ports , System.Management , System.Reflection.Context , System.Reflection.MetadataLoadContext , System.Resources.Extensions , System.Runtime.Caching , System.Runtime.Serialization.Formatters , System.Security.Cryptography.Pkcs , System.Security.Cryptography.ProtectedData , System.Security.Cryptography.Xml , System.Security.Permissions , System.ServiceModel.Syndication , System.ServiceProcess.ServiceController , System.Speech , System.Text.Encoding.CodePages , System.Text.Encodings.Web , System.Text.Json , System.Threading.AccessControl , System.Windows.Extensions , VS.Redist.Common.NetCore.SharedFramework.x64.10.0 From Version 10.0.0-alpha.1.24611.1 -> To Version 10.0.0-alpha.1.24617.3
@VSadov The symptoms of this crash are very similar to dotnet/runtime#110442 . One possible explanation is that there is a subtle race condition where we forget to unregister the dying thread and the FLS change just made it more likely to be hit. |
I was able to catch the problem under debugger. The problem is that COM apartment shutdown ends up calling into managed code and re-initializes the
|
By the looks of Are you seeing a scenario when we are called first (and detach) and then their callback gets called and they end up calling managed code? (it is not clear from the stack what calls managed code) |
Yes. |
Our assumption that a thread cannot call into managed code after calling our FLS callback is not correct then. |
Dtto for regular TLS callbacks - I guess that it is the problem in dotnet/runtime#110442. We wish that it can declared as unsupported scenario, but we have not done a good job failing deterministically in this situation, so there is likely quite a bit out code that depends on it "working". |
At high level dotnet/runtime#110442 seems similar - it also seems to be an issue with thread destruction notification. Although it is a different platform with different notification mechanism and from the discussion it appears the notification just did not arrive (or was not acted upon) before TLS was destroyed and then GC happened and crashed. Here the notification did arrive, but the thread found a way to do a reverse pinvoke after that. Unix issue feels simpler - we should be able to be notified and detach before TLS is destroyed. Something did not work as expected. The issue with thread termination callback itself entering managed code seems a bit harder to deal with. I do not have any good ideas at the moment. FLS callbacks are not very commonly used and the documentation says that they should be simple, basically just memory deallocation. Perhaps there is a way to prevent entering managed code from that? |
My guess is that the Unix issue is the same: The notification did arrive, but the thread found a way to do a reverse pinvoke after that via thread static in some native library.
We can fail-fast if the thread tries to re-enter managed code after the thread destruction notification (e.g. we can have a |
The COM FLS fallback is anything but simple. It pumps Windows messages... .
Brainstorming about possible solutions:
|
On that I agree. Poisoning a thread after detaching will get us in trouble in exactly same scenarios.
In theory it could work. Everything is synchronous as it is local to the thread, so no races to be concerned about.
If the idea is to not reregister with OS at all and just detach on every PInvokeExit, then it eliminates the concern
If registering FLS slot earlier guarantees that callback is called later (i.e. LIFO), this might be sufficient. We'd need just to be sure that the FLS is allocated before everything else when a thread is attached. (another assumption is that anything that allocates FLS slots before calling managed code will not call managed code in its callback - not the 100% robust, but plausible assumption, I think) |
It is the other way around: Callbacks registered earlier get called earlier. (As long as there is no FLS slot reuse.) We registered our FLS callback before the COM one, so it got called before the COM one. |
I think it can be the same managed/native combo. We can use the thread static callback that gets executed under loader lock to tell signal that the managed/native combo is not needed anymore (without entering the coop mode). |
Also wonder why we did not see this before. |
What exactly managed code do they need to run on apartment shutdown? Can we move that to native or fake it and not run at all (conditionally - if thread is detached) ? |
The managed code that I have seen running under WaitForApartmentShutdown was a managed Windows Proc registered by WinForms. It run because of the thread was pumping windows messages and the message happened to be for the window owned by WinForms. |
Thank you for investigating this! I'm closing this PR as #12655 has superseded this and contains the fix. |
@jkotas what was your way to repro this? Did you run a particular test or a directed repro sample? |
Build and run all winforms tests against the changes in this PR. The crash occurs when running the WinForms unit tests ( |
This pull request updates the following dependencies
From https://github.com/dotnet/runtime
Microsoft Reviewers: Open in CodeFlow